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Abstract — We analyze the (computational) complexity distri- 
bution of sphere-decoding (SD) for random infinite lattices. In 
particular, we show that under fairly general assumptions on the 
statistics of the lattice basis matrix, the tail behavior of the SD 
complexity distribution is solely determined by the inverse volume 
of a fundamental region of the underlying lattice. Particularizing 
this result to TV x M, N > M, i.i.d. Gaussian lattice basis 
matrices, we find that the corresponding complexity distribution 
is of Pareto-type with tail exponent given by N — M + 1. We 
furthermore show that this tail exponent is not improved by 
lattice-reduction, which includes layer-sorting as a special case. 

L Introduction 

The problem of finding the closest lattice point in an infinite 
lattice is commonly referred to as the closest lattice point 
(CLP) problem (see, e.g., [1]). The sphere-decoding (SD) 
algorithm [l]-[6] is a promising approach for solving the CLP 
problem. The (computational) complexity of SD, as measured 
in terms of the number of searched lattice points, depends 
strongly on the lattice basis matrix and is, in general, very 
difficult to characterize analytically. However, if the basis 
matrix is assumed random, the complexity of SD is random as 
well and one can resort to a characterization of the complexity 
distribution of SD. Previous work along these lines focused on 
the mean and the variance of SD complexity for i.i.d. Gaussian 
basis matrices [7]-[9]. However, a characterization of the tails 
of the SD complexity distribution is, for example, important 
for using SD under run-time constraints (see, e.g., [6]). In this 
paper, we make a first attempt in this direction by analyzing 
the tail behavior (TB) of the SD complexity distribution. Our 
main contributions can be summarized as follows: 

• Under fairly general assumptions on the statistics of the 
lattice basis matrix, we prove that the TB of the SD 
complexity distribution is solely determined by the TB 
of the inverse volume of a fundamental region of the 
underlying lattice. 

• Specializing this result to the case of iV x A/, N > M, 
i.i.d. circularly symmetric complex Gaussian basis matri- 
ces, we find that the complexity distribution of SD is of 
Pareto-type with tail exponent given by N — M + 1. We 
furthermore show that the tail exponent is not improved, 
i.e., increased, if lattice-reduction (see, e.g., [1]), which 
includes layer-sorting as a special case, is performed. 

This work was supported in part by the STREP project No. IST-026905 
(MASCOT) within the Sixth Framework Programme of the European Com- 



Notation: We write Aij for the entry in the ith row and jth 
column of the matrix A and Xi for the ith entry of the vector 
X. Slightly abusing common terminology, we call an x M, 
N > M, matrix A unitary if it satisfies A^A = I, where ^ 
denotes conjugate transposition and I is the identity matrix. 
The Euclidean- and the Frobenius norm are denoted by || • | 
and II • ||p, respectively, and \X\ refers to the cardinality of the 
set X. The ceil-function is denoted by [•] . Furthermore, CZ 
stands for the set of Gaussian integers, i.e., CZ = Z+-\/^Z. 
The lattice generated by the full-rank N x M (N > M) basis 
matrix A is defined as £(A) = {Ad : d e {CZ)^). For 
N — M, the corresponding covering radius is given by [10] 



/i(A) = max min ||x — Ad| 



(1) 



A circularly symmetric complex Gaussian random variable 
(RV) with variance is denoted as a; ^ CA/'(0,ct^). 
The natural logarithm is referred to as log( ). We write 
g{x) = f{x), X ~* xn, for lim^^^„ log5(a;)/log(a;) = 
limjj^-cg log /(x)/log(a;), assuming that the correspond- 
ing limits exist. The symbols < and > are defined 
analogously. Finally, non-polynomial behavior of g{x) 
is captured by lim^^-^o^olog (7(a;)/log(x) = ±oo or 

0, for which we write g{x) = 



iirax^xo^oS9{x)/log{x) 



„±oo 



, X ■ 



xq, and g{x) 



x'^, X ■ 



Xq, respectively. 



A. Sphere-Decoding 

The CLP problem refers to computing 



d — arg min ||r — 

de(cz)" 



Hd||2 



(2) 



for a given vector re and a given full-rank matrix 
H G £NxM^ > words, solving amounts 

to finding the point in the lattice £(H) that is closest (in 
Euclidean distance) to r. In communications, (|2| is known as 
the maximum-likelihood (ML) detection problem for detecting 
d' e (CZ)^^ based on the hnear model r = Hd' + w with H 
known at the receiver and w being i.i.d. circularly symmetric 
complex Gaussian noise. 

A prominent approach for solving (|2]) is the SD algorithm 
[l]-[6]. In the following, we consider Fincke-Pohst SD [2] 
without radius reduction (see, e.g., [7]). The algorithm starts 
by computing the (unique) QR-decomposition (QRD) H — 
QR, where R denotes an M x M upper triangular matrix 



with positive real-valued elements on its main diagonal and 
Q of size X Af is unitary. Then, (|2]l can equivalently be 
written as 

d = arg min ||y - Rd|p (3) 

de(cz)'" 

where y = Q^r. Next, (|3]l is solved subject to a sphere 
constraint (SC), which amounts to considering only those 
d e (CZ)^^ that lie within a hypersphere of radius p around 
y, i.e., all d that satisfy 



Rd||2 < p^. 



(4) 



Here, the sphere radius p has to be chosen sufficiently large 
for the search sphere to contain at least one lattice point Rd. 
Note, however, that if p is chosen too large, too many points 
will satisfy the SC and the complexity of SD will be high. 
As detailed next, imposing a SC enables an efficient recursive 
solution of the triangularized CLP problem ([3]l. 

Consider the length-A; subvectors d^: G (CZ)*^' of d defined 
as dfe = (dM-fe+i • • • (Im)^ , k — 1, . . . , M, where fc is a 



layer index. The metric ||y — Rd|p 
be computed recursively according to 



llYM 



RAfd 



Rfcd 



fcUfel 



= ||yfe-i - Rfe_idfe_i| 



|Afc(dfe)|2 (5) 



where Afe(dfe) = yn-k+i - Y^f^M-k+i^M-k+iAdt, Rfe 
denotes the fc x fc bottom right (upper triangular) submatrix of 
R associated with d^, and y^ = {yi\j^k+i ■ ■ ■ Um)^ ■ Thus, 
with (|5]l, a necessary condition for d to satisfy the SC is that 
any associated d^ satisfies the partial SC 



lYk 



(6) 



This formulation now enables finding all integer vectors d that 
satisfy Q in an efficient (recursive) manner as detailed, e.g., 
in [5], [7]. 

II. Complexity Distribution of SD 

We define the computational complexity of SD as the 
number of lattice points searched by the algorithm, i.e., the 
number of vectors dfe e (CZ)'', fc = 1, . . . ,M, that satisfy 
the partial SCs in (|6]l (cf. [7], [11]). Specifically, we define 
the fcth layer complexity of SD as 



^fe-|{dfee(CZ)'=: ||yfe-Rfedfe||2 <p2}| 



(7) 



with the total complexity S — X^feli ^k- It was shown in [5] 
- for the finite lattice case - that S is proportional to the run- 
time complexity of a corresponding VLSI implementation. 

A. Complexity Distribution and Tail Exponents 

The quantities S'fc, fc = 1,...,M, and S, defined above, 
are functions of H, r, and p. In the following, we let H 
and r be random (potentially statistically dependent) and 
consider a fixed p that does not depend on the realizations 
of H and r. For example, in the case of ML detection of 
the transmitted data vector d' in MIMO wireless systems, 
r = Hd' + w, where the entries of H (the channel matrix) 
and w (the noise vector) are typically assumed i.i.d. circularly 
symmetric complex Gaussian. In this setting, a reasonable 



fixed choice of p can be based on the noise statistics such that 
the probability of finding the transmitted data vector inside the 
search hypersphere is sufficiently high (see, e.g., [7]). Note, 
however, that this results in a nonzero probabiUty of failing to 
find an integer vector inside the search hypersphere, which, in 
the absence of restarting the search with a larger sphere radius 
(see, e.g., [7]), would entail an error floor. 

Since both H and r are random, Sk and S are random 
as well and can be characterized through their respective 
distribution functions P[Sk > L] and P[S > L]. While these 
distributions seem hard to come by analytically, it turns out 
that the corresponding tail exponents ^fe, fc — 1, . . . , M, and 
^, defined by 



nsk > L] 



L 



oo 



and 



P[S >L] = L-^, L 



are amenable to an analytical characterization. We note that 



S = J2k=i ^k impUes 

^ = min{^i,f2, • • • ,6/}- 



(8) 



The tail exponents characterize the TB of the corresponding 
complexity distributions in terms of polynomial decay rates in 
L for L — > oo. We note that for finite (non-zero) tail exponents, 
the corresponding complexity distributions are of Pareto-type 
meaning that they decay polynomially in L. In particular, if the 
complexity distribution P[S' > L] has tail exponent ^, one can 
state that L-(«+*') < P[S > i] < L^^^"*) for any 5 > 
and sufficiently large L. Furthermore, if P[5'^^ > L] and 
P[S'(2) > L] have tail exponents ^(i) and ^(2) with ^(i) > 
we can conclude that P[S'(i) > L] < P[S^^^ > L] for 
sufficiently large L. Hence, larger tail exponents are desirable 
since this implies that the probability of the complexity being 
atypically large is smaller. This, for example, is advantageous 
in the context of MIMO detection under run-time constraints 
(i.e., under limits on the number of lattice points that can 
be searched). We emphasize, however, that the complexity 
tail exponents, as defined above, do not capture multiplicative 
constants and do not characterize the small L behavior of the 
corresponding complexity distributions. 

B. Main Result 

The complexity of SD can often be reduced by employing 
preprocessing techniques such as lattice-reduction (LR) or 
layer-sorting (LS) (see, e.g., [1]). In the remainder of this 
paper, we account for preprocessing by assuming that y is 
a general function of r and H and R is a general function of 
H. For example, the direct QRD H = QR (see Section I-Ai 
results in the special case y = Q^r and R = Q^H. 

Theorem 1: Consider SD with fixed p {Q < p < oo) and 
let H and r be random (potentially statistically dependent). 
The corresponding fcth layer complexity Sk, defined in (|7]i, 
satisfies 



P[Sk >L\=P 



1 



> L 



_det(RfRfe) 
if all of the following conditions are met 



(9) 



• Statistics of H: The probability density function (pdf) 
/(H) of H satisfies the scaling property 

/(H) > /?/(aH) (10) 

for all H e C^^*^ and all a e M, a > 1, with some 
constant /3 e M, ^ > 0. 

• Statistics of H and preprocessing: The covering radius 
n(R.) of £(R) satisfies 

P[fi{R.) > L] ^ L-°° , L^oo. (11) 

• Preprocessing: Let det(R^Rfc) = 9fe(H) with gk : 

IR+. The functions gfe(H) and (7^(H) satisfy, respectively, 
the scaling properties 

gkibU) = 5"-fffc(H) (12) 

and 

g^ibU) = fe".g^(H) (13) 

for all H e C^^*^ and all 6 G M, & > 0, with some 
constants a^, a G M, a^. > 0, a > 0. 
Proof: See Appendix. □ 

Discussion: Theorem 1 states that the TB of P[Sk > L] 
is fully characterized by the TB of P[l/det(Rf R^) > L] 
provided the conditions ([TO]l - ( [T3] l are satisfied. It is immediate 
that the TB of P\Sk > L] then depends neither on the statistics 
of r nor on the particular fixed choice of p. Consequently, r 
and p will influence P[5fe > L] (for example, a larger p will 
certainly result in a larger value of P[Sk > L]) but do not 
affect the corresponding complexity tail exponent. The condi- 
tions ([TO]l - ([T3]l constitute fairly general requirements on the 
statistics of the lattice basis matrix H and on the preprocessing 
algorithm. For example, for direct QRD or conventional LR 
(see, e.g., [1]), it can be shown [12] that all the conditions 
above are satisfied if the entries of H are jointly Gaussian- 
distributed with arbitrary non-singular covariance matrix and 
arbitrary finite mean, i.e., for H being a Rayleigh- or Ricean- 
fading MIMO channel with (non-singular) covariance matrix. 

It is interesting to note that det(R^R/j) is the volume of 
a fundamental region of £(Rfc) [10]. A well-known approxi- 
mation for Sk is given by [13] 

a Mr) 



where 



Vk{p) 



det(RfRfe) 

7r''(p^)fe 
fc! 



(14) 



is the volume of a hypersphere in k complex-valued di- 
mensions. This approximation simply counts the number of 
fundamental regions (each occupied by exactly one lattice 
point) that fit into the fc-dimensional search sphere and be- 
comes exact if an averaging of is performed over 
uniformly distributed over a fundamental region of £(Rfe) 
[13]. Motivated by this result, Sk has been used in [1] and [11] 
to assess the complexity of various SD variants. For the TB, 
it immediately follows that (|9]l can equivalently be written as 
^[Sk > i] = P \Sk ^ L\, L ^ oo, and no averaging argument 
is required. 



III. Tail Exponents for i.i.d. Gaussian H 

Specializing Theorem 1 to lattice basis matrices H whose 
entries are i.i.d. CA/'(0,cr^) (the model typically used in 
MIMO detection) leads to particularly interesting results. In 
this case, the pdf of H is given by /(H) = ci e^'^^H^"'' 
with some constants ci , C2 > 0, which directly implies that 
condition ( [TO] i is satisfied with j3 — \. 

A. Tail Exponents for Direct QRD 

For direct QRD of H (i.e., R is obtained through H — 
QR), /i'^(R) can be upper-bounded as (see, e.g., [11, 
Prop. 1] extended to the complex-valued case) /i^(R) < 
\ HiLi Rli = z"^- It foflows fr om [14, Lemma 2.1] that z is 
a x-distributed RV, which, upon noting that P [/i(R) > L] < 
P[z > L] implies that condition ( [TT| is satisfied (see [12] 
for a detailed proof of this statement). Condition ( [T2j i is 
verified by observing that the QRD of 6H results in &R, 
which gives gk{hii) — b'^''gk(H.)- Condition ( [T3| ) is shown 
to be satisfied by noting that ([T]) implies /i(6R) = 6/i(R) 
and hence g^(6H) — 6g^(H). Therefore, all the conditions of 
Theorem 1 are met. Finally, using results from [15], the TB 
of the distributions of the layer complexities of SD for direct 
QRD and i.i.d. Gaussian H can be established as [12] 



P[Sk >L]=L 



^ T-{N-M+1) 



L 



oo, 



fc = 1, 



and, consequently, 

P[5>L] = L-(^-*^+i\ L 



,M (15) 



(16) 



We conclude that the distributions of the layer and total 
complexities are of Pareto-type with tail exponents £,k = £, = 
N — M + 1, fc = 1, ■ • • , M. These results show that increasing 
N (e.g., the number of receive antennas in a MIMO context) 
for given M (e.g., the number of transmit antennas) results in 
improved tail exponents. 

B. Tail Exponents for LR-Based Preprocessing 

We define LR-based preprocessing (see, e.g., [1]) as apply- 
ing, prior to the QRD, the transformation H — HT, where T 
is an M x M unimodular matrix, i.e., Ti,j e CZ, Vi,j, and 
|det(T)| = 1. The matrix T is obtained, for example, through 
the LLL algorithm [16], which finds a basis matrix H of the 
lattice £(H) that is "closer" than H to an orthogonal matrix. 
Another important preprocessing technique is LS (e.g., with 
the V-BLAST algorithm [17]), which is just a special case of 
LR obtained by restricting T to be a permutation matrix. 

The triangularized form of the CLP problem based on LR 
preprocessing is given by (|3]l with R and y Replaced by R 
and y ~ Q^r, respectively, where Q and R are the QR- 
factors of H, i.e., H = QR. If we denote the corresponding 
solution of ([3| as d, the final solution of (|2]) is^ d = Td. We 
now consider Theorem 1 with R replaced by R. Let us write 
RT = Q' R;^ where Q' and R' are the QR-factors of RT. 
Noting that H = QRT = QR, we obtain QQ' R' = QR. 
Since QQ' is unitary and the QR-factors are unique, it follows 
that QQ' = Q and R' = R, which implies 



R = Q' RT 



(17) 



As T^^ is unimodular (since T is unimodular) and Q' is 
unitary, it can be verified that /i(R) = A'(R-) and det(R^R) = 
det(R^R). Due to /i(R) = m(R.) and the results for direct 
QRD in Section |III-A[ conditions ( [TTI and ( [T3] l are satisfied 
for LR-based preprocessing. 

LRfor k = M: Due to det(R^R) = det(R^R), condition 
( [T2| l is satisfied for k = M and LR-based preprocessing. Now, 
applying Theorem 1, we can immediately conclude that LR- 
based processing results in the same Mth layer complexity 
tail exponent as direct QRD, i.e.. 



10" 



P[Sm>L] = L-<-^-^'+'\ L 



(18) 



or, equivalently, £^m = N — AI + 1. From ([8]l, we can therefore 
conclude that ^ < — A/ + 1 for LR-based preprocessing, 
which shows that LR (including LS) does not improve (i.e., 
increase) the total complexity tail exponent as compared to 
that obtained for direct QRD. 

LRfor k < M: It can be shown [12] that all LR algorithms 
delivering a unimodular transformation matrix T, which is 
invariant to a positive scaling of H, i.e., H and bH for all b E 
R, b > 0, result in the same T, satisfy condition ([12]) for k = 
1, . . . , AI. We note that this is the case for any reasonable LR 
algorithm we are aware of. Prominent examples are the LLL 
algorithm [16] and LS according to the V-BLAST algorithm 
[17]. Here, Theorem 1 therefore implies that 



P[Sk >L]=P 



1 



det(RfRfe) 



> L 



L 



(19) 



for fc= l,...,Af. 

LR Based on LLL: For LR carried out through the LLL 
algorithm [16] (see [18] for its complex-valued extension), 
based on ([T9]l, one can show the more specific result [12] 



P[5fe >L\<L- 



k^l. 



or, equivalently, ^j. > N/ k. Compared with = N — M + 1 
for direct QRD (cf. ([T5|)), we can conclude that LLL pre- 
processing improves (i.e., increases) the tail exponents up to 
layer k < \N/{N - Af + 1)] - 1. In the following, consider 
N = M. We have = M/k > 1, fc = 1, . . . , Af - 1, and 
£,M = 1 (see ([T8]l), which, in this case, establishes that the 
TB of the distribution of the total complexity of SD with LLL 
preprocessing is dominated by the TB of the distribution of 
the A/th layer complexity; in particular, we have ^ = = 1, 
as in the case of direct QRD (cf. ([T6|). 

IV. Numerical Results 

We consider SD for data detection in A^x Af MIMO wireless 
systems, where r = Hd' + w with the entries of H and w 
assumed i.i.d. C7V(0, l/AI) and i.i.d. C7V(0, cr^), respectively, 
and with the transmitted vector d' £ (CZ)^. We choose the 
radius /? in Q such that d' is found by the SD algorithm with 
probability 0.99 and for l/cr^ we assume a value of 15 dB. 
We note that the complexity of SD is random in H and w 
and does not depend on d'. 

Fig. [T] shows the distribution of the total complexity P[S' > 
L] in double log-scale for SD with direct QRD, V-BLAST 
LS [17], and with (complex-valued) LLL preprocessing [18, 



AI 
Oh 




-e— direet QRD 
--a-- V-BLAST LS 
-isr- LLL 



Fig. I. Distribution of total complexity P[5 > L] of SD with direct QRD, 
V-BLAST LS, and LLL preprocessing for 4 X 4 and 5x4 MIMO systems. 



with pai-ameter (5 = 3/4] for 4 x 4 and 5x4 MIMO systems. 
We can see that the results reflect our analytic findings. For 
example, for direct QRD in the 4x4 case, the distribution 
of the total complexity in Fig. [T] exhibits a large-L behavior 
of L^^ as predicted by ([T6| for N = M. Furthermore, it can 
be seen that adding one receive antenna, indeed, improves the 
TB and leads to a large-L behavior of L^^ (cf. ([T6|l). Finally, 
the numerical results indicate that LLL preprocessing and V- 
BLAST LS do reduce the complexity, as compared to direct 
QRD, but do not improve the tail exponents. 
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Appendix 
Proof of Theorem 1 

The proof of Theorem 1 is based on separately es- 



tablishing the exponential lower bound P[Sk 



> 



L] 



> 



P[l/det(R|'^Rfc) > L], L ^ oo, and the exponential upper 
bound P[Sk > L] < P[l/det(Rf R^) > i], i oo, which 
then combine to P[Sk > L] = P[l/det(Rf R^) > L]. 

A. Exponential Lower Bound 

We start by noting that [13, Ch. 3.2, Eq. (3.3)] 



Sk > 



det(RfRfc 



where Vk{p) and Ak{p) denote the volume (cf. ( [T4] l) and the 
surface area of the search sphere at layer k, respectively]^ 
Using ^(Rfe) < ^(R), fc = 1, . . . , Af , [12], we obtain 



P[Sk >L]>P 



Vkip) - piR)Ak{p) 



> L 



det(RfRfe) 

Consider a constant c € M, c > 0, such that Vk{p) — cAk{p) > 
and define c' — Vk{p) — cA^ip) > 0. We then have 



P[Sk > > P [H e 6] 



(20) 



'Note that condition implies full-rank Rj. with probability one. In 
particular, det(R^Rfe) > and ^(R^) < oo with probability one. However, 
it is straightforward to show that Theorem 1 also holds in the case where Rfc 
is rank-deficient with non-zero probability, which leads to P[5ft > L] = 
P[l/det(Rf Rfc) > L] = L°, oo. 



where B = {H: > i) n (.g^(H) < c)} with 

det(Rf Rfc) = .gfc(H) and ^(R) = 5^(H). With property 
( [TO| l, we further obtain 

P[H eB]= [ /(H)dH > /? / /(L''H)dH 

for all (5 > 0, L > 1, and some f] > 0. Performing the change 
of variables H' = L^H and invoking conditions ( [T2] l and ( [T3| ) 
yields 

P[H eB]>p L-^^'^^ P[H e 6'] (21) 



where 

S'= <!H: 



n (5^(H) < cL'^) 



Next, noting that for two events Ai and A2, by the inclusion- 
exclusion principle, P[Ai n A2] > P[^i] - ^[M], where A2 
denotes the complementary event of A2, we get 



P[H e 6'] > P 



5fc(H) 



> L 



P[g^(H) >cL 



(5al 



Now ( [TT] ) with ^(R) = and 5, a > impHes 

Pjg^(H) > cL^""] = L-°°, L ^ 00, which, together with 
Q and (|2T), yields 



P[5fc > L] > L- 



-2MNS 1 



5fe(H) 



> L 



00. 



Let us write P[l/gfe(H) > = i i ^ 00, for some con- 
stant a > 0. We then have ?[Sk >L\> L-^MNS-(i-Sa^)a ^ 
L ^ 00. As this result holds for arbitrarily small values of S, 
we can conclude that P[Sk > L] > L'" = P[l/ gk{H.) > L], 
L — > 00, which establishes the exponential lower bound. 

B. Exponential Upper Bound 
From [13, Ch. 3.2, Eq. (3.3)] 

det(RfR,.) 

which, again using pCRk) < /^(R), k — 1, . . . , M, results in 

Vfe(p + /i(R)) 



P[Sk >L]<P 



det(RfR,) 



> L 



(22) 



Note that P[xy > L] = P[{xy > L) n {y < L^)] + 
P[{xy >L)n{y> L^)] < P[x > L^-^] + P[y > L^] for 
any two RVs x,y E M. and any constant (5gM, 0<(5<1. 
Applying this to ( |22] i with x — l/det(R|'^Rfc) and y — 
Vk{p + p{R)), we get 



P[Sk >L]<P 



> L 



1-5 



_det(RfRfc) 

+ p[T4(p + m(R)) >i*] 

With ( [l4l ) and the binomial theorem, we can write 

, .r'= ^ /2fc\ 



which, using P 
set of RVs {x,}fl-^ yields 



Efii x^>L < j:fi^ P[x, > L/M] for any 



2k 

p[Vk{p+p{R.))>L'] <Y,p[wy>L' 



i=0 



L 



00. 



Property ( fTTj l (for the terms corresponding to i > 0) and 

P[c" > i] < e"(^"=") = L-°°, L ^ 00, for any constant 
c" > (for the term corresponding to i = 0) now directly 
imply P[Vk{p + Ai(R)) > L^] = L 00, and, hence. 



P[Sk >L]<P 



1 



> L 



00. 



00, 



_det(RfRfc) 

As before, writing P[l/det(Rf R^) > i] = L-", L ^ 
for some constant a > 0, we get P[Sk > L] < ^-(i-*)'^, L 
00. As this result holds for arbitrarily small values of 6, we can 
conclude that P[Sk > L] < L-" = P[l/det(Rf R^) > L], 
L — > 00, which establishes the exponential upper bound. 
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